Skip to content

fix(ci): pass --shard to vitest in the windows e2e matrix [RED-738] [ship]#1415

Merged
sorccu merged 1 commit into
mainfrom
simo/red-738-e2e-windows-shard-flag
Jul 17, 2026
Merged

fix(ci): pass --shard to vitest in the windows e2e matrix [RED-738] [ship]#1415
sorccu merged 1 commit into
mainfrom
simo/red-738-e2e-windows-shard-flag

Conversation

@sorccu

@sorccu sorccu commented Jul 17, 2026

Copy link
Copy Markdown
Member

Linear: RED-738

The e2e - checkly - windows job declares a 4-way shard matrix (1/44/4), but all four shards ran the entire 27-file e2e suite — the matrix bought nothing.

Root cause

Not the compound test:e2e script, which is what the symptom suggests. Args appended to A && B land at the end of B, and B is the vitest invocation, so they reach vitest fine.

The sole cause is the literal --. pnpm forwards it verbatim into the script (npm strips it), where vitest's parser collects everything after it into an argument list it never reads — so --shard is silently discarded rather than reinterpreted. That is exactly why the broken form ran green and complete instead of failing.

Verified on pnpm 10.34.1, the version CI uses:

Invocation ARGV the final command receives
pnpm --filter thing run compound -- --shard=1/4 ["--", "--shard=1/4"]
pnpm --filter thing run compound --shard=1/4 ["--shard=1/4"]

pnpm does not intercept flags after the script name — even --reporter, which pnpm itself defines — so no separator is needed.

Also ruled out: passing the shard through an env var read by the e2e vitest config. Vitest's --shard is CLI-only and has no config-file equivalent.

Verification

A green run proves nothing here — the broken form passed too, running the full suite four times over. Two checks were done locally:

  1. pnpm now echoes pnpm pack && cross-env NODE_CONFIG_DIR=./e2e/config vitest --run -c ./vitest.config.e2e.mts --shard=1/4, with no -- before the flag.
  2. vitest list per shard yields a true partition — 27 unique files, no overlap, no omission:
Shard Files Notable
1/4 7
2/4 7
3/4 7 test.spec.ts
4/4 6 deploy.spec.ts

To confirm on this PR: the four e2e - checkly - windows (n/4) logs should report file counts smaller than 27 summing to 27 (7/7/7/6). Shards 1–3 all reporting 7 is expected and correct — the discriminator is 7+7+7+6=27 against the broken 27+27+27+27=108.

What this does and does not buy

The ticket originally claimed ~4x on both time and load. More precisely:

  • Total API volume against the shared test account drops ~4x (108 file-runs → 27), shortening the window it spends under load.
  • Peak concurrent requests are unchanged. The e2e vitest config sets no fileParallelism/maxWorkers, so ~4 files already run at once per job: peak is ~4 shards × ~4 forks either way. If the intermittent HTTP 500s from the test-session endpoint persist, that is not evidence this flag failed to land.
  • The real hot-spot win: all four shards previously ran deploy.spec.ts concurrently, each doing an account-wide cleanupProjects() sweep. Now only one does — 5 concurrent copies (4 Windows + Ubuntu) down to 2.
  • Wall-clock is bounded by the heaviest shard, not total/4. deploy.spec.ts sets the floor, and globalSetup doesn't shard: every shard still pays pnpm pack plus three template installs regardless of how few files it runs.

Risk

This is the first time the suite has actually been split, so latent cross-file coupling surfaces here for the first time.

No ordering coupling exists: every mutating spec is self-isolating via uuidv4()/nanoid() IDs, and vitest already runs files in parallel forks with no ordering guarantee. But checks-list.spec.ts, checks-get.spec.ts and checks-stats.spec.ts assert on account-wide check counts and now run in shards without deploy.spec.ts populating the account. Failures isolated to those three specs indicate that assumption, not the flag. Rollback is a one-character revert.

Other changes

None — single-file change.

The `e2e - checkly - windows` job declares a 4-way shard matrix, but all
four shards ran the entire 27-file e2e suite.

pnpm forwards a literal `--` verbatim into the script (npm strips it),
where vitest's parser collects everything after it into an argument list
it never reads — silently discarding --shard. Dropping the separator is
sufficient: pnpm appends trailing args to the script command as-is, so
--shard reaches vitest without one.

The compound `test:e2e` script is not the cause. Args appended to `A && B`
land at the end of B, and B is the vitest invocation.

Verified locally that the four shards now partition the suite 7/7/7/6 into
27 unique files. A green run is not evidence on its own — the broken form
passed too, running the full suite four times over.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sorccu sorccu changed the title fix(ci): pass --shard to vitest in the windows e2e matrix [RED-738] fix(ci): pass --shard to vitest in the windows e2e matrix [RED-738] [ship] Jul 17, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: ship/show PR from a same-repo branch.

@sorccu
sorccu merged commit 23abb3d into main Jul 17, 2026
36 of 42 checks passed
@sorccu
sorccu deleted the simo/red-738-e2e-windows-shard-flag branch July 17, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant